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1. Background 

Paramax Space Systems began its mission in 1986 as a member of the Rockwell Space 
Operations Company (RSOC) team which was the successful bidder on a massive operations 
consolidation contract for the Mission Operations Directorate (MOD) at the Johnson Space 
Center (JSC). The contract awarded to the team was the Space Transportation System 
Operations Contract (STSOC). Our initial work force consisted of less than twenty people. We 
staffed the contract with employees from the unsuccessful incumbent contractors. Our initial 
challenge was to accept responsibility for a very large, highly complex and fragmented collection 
of software from eleven different contractors and transform it into a coherent, operation 
baseline. Concurrently, we had to integrate a diverse group of people from eleven different 
companies into a single, cohesive team. Paramax executives recognized the absolute necessity 
to develop a business culture based on the concept of employee involvement to execute and 
improve the complex processes of our new environment. Our executives clearly understood t at 
management needed to set the example and lead the way to quality improvement. One of our 
first acts as a management team was to develop, document and display our quality policy, which 

states: 


• We shall strive for excellence in all endeavors. 

• We shall set our goals to deliver error-free products and services on time. 

• We shall understand and conform to the requirements. 

• We shall understand the software processes associated with our jobs. 

• We shall measure our performance in terms of satisfying the requirements. 

• We shall analyze failures and take corrective action to prevent their recurrence. 


All executives and managers signed this pledge. It is posted throughout the various buddings, 
including several at JSC, where our employees work. 

2. Paramax Space Systems Operation Spaceflight Role 

Paramax Space Systems supplies $78 million of software products and services annually for the 
Space Shuttle and Space Station programs at the Johnson Space Center and projects at the GSFC. 
Our current major projects include the following. 

• Space Shuttle and Space Station Freedom operations 

• Mission Control Center upgrades 

• Space Station Control Center development 

• Software product and quality assurance at JSC 
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• Space Shuttle preflight evaluation 

• Information Systems support 

• Quality engineering and quality assurance at GSFC 

Space Systems manages, modifies and maintains all ground-based software for the Space Shuttle 
program. We provide products and services for all phases of Shuttle operations, including flight 
design and mission planning; astronaut and flight controller training; preflight and postflight 
verification of orbiter software, systems and components; and real-time command, control and 
communications in JSC’s Mission Control Center. 

This is, to the best of our knowledge, the largest, most complex scientific/technical software 
project in the world. Our work at JSC encompasses 19 million lines of code written in 15 
programming languages and running on more than 300 computers in 13 facilities. This is truly 
a massive task. 

At the Goddard Space Flight Center, we provide quality engineering and quality assurance for 
Earth-observing and scientific satellites. We also evaluate the safety, reliability and quality of 
electronic, mechanical and other components, and calibrate and repair testing equipment. 

Paramax Space Systems employs more that 1 ,000 people at its Houston location and more that 
250 people in Lanham, Maryland. 

3. Focus of our Quality Management Effort 

Our commitment to the quality process is articulated in our quality policy. Our commitment to 
our customers is to achieve increasing levels of reliability, productivity and responsiveness. 
The software we maintain is a critical component in the safety and mission success of Space 
Shuttle missions. Safety, of course, is our first concern. We continually strive to improve the 
productivity of our software engineering processes to support NASA’s worthy goals to establish 
a permanent presence in space, on the Moon and on Mars. The nature of manned spaceflight 
demands our immediate response to identify and correct failures, and ensure they never recur. 

We achieve our goals through the commitment of our management team to the quality process, 
their emphasis on involving all of our employees in improvement teams, and the use of metrics 
and measurements to manage our business. 

4. Management Commitment 

Paramax management is visibly and actively involved in every aspect of our quality process. 
Managers and executives provide leadership through our quality infrastructure, participate in our 
quality education process, and work hand-in-hand with our Excellence Teams to foster total 
organizational involvement and cross-functional teamwork. Our managers address quality topics 
as a regular agenda item in staff meetings to ensure ongoing awareness of the need for 
continuous improvement. Paramax also establishes quality improvement goals as an integral part 
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of our annual strategic planning process. All our annual organizational goals have a quality 
orientation. Quality goals are included in each manager’s annual performance plan, and we are 
beginning to establish role model standards to measure and improve our leadership processes. 

5. Employee Involvement through Team Excellence 

Management commitment is the foundation of our quality effort, but significant improvement 
is not possible without the active involvement of all employees. At Par*tmax, this is 
accomplished through the Team Excellence process. 

Our work force is divided into functional Centers of Excellence, each represented by an 
Excellence Team composed of management and employees. Excellence Teams represent work 
groups at every level of the organization. Each Excellence Team is required to. 

• identify and document the processes under their direct control; 

• establish metrics and measurements to monitor the processes; 

• initiate corrective action and process improvement; and 

• publicly post the results on the team bulletin board in the team work area. 

Initially, Team Excellence board requirements were established by the Paramax Quality 
Improvement Team and included the following elements. 

• Team Mission Statement 

• Team Goals 

• Procedure Reference 

• Team Members 

• Visitor’s Log 

As this activity evolved, teams developed more sophisticated Excellence Board criteria. 
Excellence Teams are currently evaluated on a quarterly basis according to specific criteria. The 
purpose of the evaluation is to ensure the continued effectiveness of Excellence Teams. Special 
recognition is provided for teams who achieve a perfect score on the evaluation, and all teams 
are honored annually for their contributions to the organization. The Team Excellence bulletin 
boards have created a "window" into the daily operation of each team and have involved every 
employee in the improvement process. The boards are a forum for each team to showcase their 
efforts and accomplishments. Team goals are publicly displayed, as are the accompanying 
process metrics. Examples of Team Excellence boards are depicted in figure 1 through figure 

5. 
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Figure 2 - Team Excellence Secondary Metrics Board 
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Figure 3 - Team Excellence Process Improvement Board 
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Figure 4 - Team Excellence Recognition Board 
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6. The Measurement Process 

The significant successes we have enjoyed are direct results of our quality efforts. A major 
element of these achievements is our emphasis on a data-driven, decision-making process. We 
use a structured metrics development and deployment methodology to manage all our processes. 

The initial step in an effective metrics development effort is to define the process. Until and 
unless a process has been defined, analyzed and documented, metrics and measurements can not 
be applied to monitor and improve it. Once process definition has been completed, a decision 
must be made concerning the process area most in need of improvement. The individuals and 
teams who work closely with the process, the implementors, are best suited to determine how 
to measure the performance of the process. At this point, we are interested in performance of 
the process rather than the quality of the end product. While product quality is always 
significant, our goal in process analysis and measurement is first to improve the process itself; 
improvement in the end product win naturally follow. 

The next step is to measure the process and establish an adequate data baseline. Several periods 
must be measured to ensure that a descriptive trend develops. Once the data has been measured 
and baselined, it must be evaluated to determine the state of the process. At this point, 
improvement objectives can be defined and action can be taken to improve the process. 
Decisions must be made by the team who implemented the process, the process improvement 
team (if another team) and management about improvement priorities. Pareto analysis, cost- 
benefit analysis and decision-support tools must be applied to ensure that scarce resources are 
effectively used in the improvement process. None of the tools can be applied until the state of 
process has been determined. Figure 6 provides an example of reevaluating commitments based 
on collecting and analyzing metrics. Once a process has been optimized, it may still be 
necessary to monitor progress to validate long-term stability. 

7. Measuring Performance 

When we establish metrics, we ensure they are easily collectible, unambiguous, meaningful, 
important, controllable and representative of the process being monitored. If any of these 
requirements is violated, it is necessary to evaluate the metrics to determine their relationship 
to the improvement objectives for the process. It is sometimes necessary to use a different 
measurement technique or different metrics to meet the defined improvement goals. 


8. Standard Metrics 

We have established standard metrics for each organizational element and Excellence Team. 
The metrics are collected in four categories. 

• Quality Performance 

• Workflow Performance 

• Productivity Performance 
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• Team-Building 

The standard metrics set facilitates consistent communication within the organization and gives 
Paramax management a regular summary as well as long-term trend performance information 
about individual teams or the organization as a whole. The metrics are communicated through 
the organization by using the objectives matrix. Paramax standard metrics are displayed in 
figure 7. 

9. The Objectives Matrix 

The objectives matrix is a tool for summarizing measurement objectives and accomplishments. 
It is a method to record current performance, goals, the proportionate value of performance 
indicators and the rate at which improvement is made. The data can then be analyzed to 
determine trends and concentrate attention on areas requiring improvement. The objectives 
matrix can be used to measure a project, program or organization and can be rolled up to 
higher-level matrices. This is an exceptional tool to understand and monitor the health and 
status of an organization. An objectives matrix is displayed in figure 7-1. 
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1991 Performance on New Commitments 


Criteria 


Performance Goal 


Satisfiers 


SR closure to receipt ratio 

DR closure to receipt ratio 

Milestones met 

Labor hours per SR 

Resource allocation for SRs 

Resource allocation for Engineering 

Engineering productivity (KLOC per Engineer) 

Monthly process improvements 

Team effectiveness 

Training hours/employee/month 


Dissatisfiers 


Range 


.99 

1.0 

1.01-1.09 

1.0 

87-99% 

100% 

232 - 358 

170 

24 - 25.5% 

32% 

50.0 - 53.3% 

64% 

40.0-41.8 

44 

79-122 

76 

99.23 - 100.07% 

100% 

2.0 - 2.58 

4.0 


SR backlog 
DR backlog 
Process failures 

Mean-time-to-fix, critical problems 
Mean-time-to-fix, all problems 
Attrition rate 


1283 - 1314 

<1000 

1290 - 1342 

<1000 

126 - 129 

0 

18 * 20 days 

13 
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30 

5.17-7.78% 

< 9.5% 


M 


Figure 6 
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Figure 7 - SSO Standard Metrics 
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ORGANIZATION: PARAMAX • Spec* Syeteme 
John B. Munson 

Software Sustaining Engineering 
Perfomance 

DATE: October, 1992 
Director, Technical E 

Ixcelleno 

e: C.B. Ds 

nlele 

Criteria 

How Measured 

TRIa 

OBJECTIVES MATRIX 

Performance Levels 

Score 

4 

Weight 

1 

20 

Value 

80 

inis 

Month 


1 

2 


1 

5 

6 

7 

8 

9 

10 

OR Closure 
Index 

Closure/Receipt Ratio 

1.06 

.85 

.90 

• 

.95 

B 


1.10 

1.15 

1.20 

1.25 

1.30 

1.35 

SR Closure 
Index 

Closure/Recelpt Ratio 

1.60 

B 

.85 

.90 

.95 

H 

1.05 

1.10 

1.15 

1.20 

• 

1.25 


10 

20 

200 

Milestones 

Mel 

No. 1st Level 
Milestones Met (%) 

97.9 

90 

91 

92 

93 

94 

95 

96 

© 

• 

98 

99 

100 

7 

15 

105 

SR 

Backlog 

EOM Open SRs 

900 

1500 

1400 

1300 

1200 

1100 

• 

100 


800 

700 

600 

500 

6 

15 

90 

DR 

Backlog 

EOM Open DRs 

993 

1500 

1400 

• 

1300 

1200 

1100 


900 

800 

700 

600 

500 

5 

10 

50 

Critical DR 
Responsiveness 

Mean Time to Fix 
Critical DRs 

14 

25 

23 

21 

19 

17 

IS 

13 

• 

11 

9 

7 

5 

6 

10 

50 

Critical DR 
Age 

Average Age of EOM 
Open Critical DRs 

0 

34 

33 

32 

31 

■ 

29 

28 

27 

26 

25 

6 

10 

10 

50 




B 

B 

■ 

B 

■ 













■ 


■ 

fl 

■ 













■ 

■ 

■ 

B 

■ 










Circle Current Month's Score Q 

Solid dot for Prior Month's Score • 

Current Monlh 

625 

Previous Month 

£Q5_ 

Variance 

♦ 20 


Paramax Systems Corporation 


Figure 7-1 

































15 


10. Organizational Results 

Our Team Excellence program promotes continual evaluation and enhancement of our software 
engineering processes. The effectiveness of this program is indicated by the major 
accomplishments achieved by Paramax since 1989; some of which are listed below. 

• Software engineering productivity has been increased 41 percent, a savings equivalent 
to 140 full-time employees. 

• More than 3900 modifications of the software baseline have been implemented. 

• More that 850 modifications of the software baseline have been initiated. 

• More than 5 million additional source lines of code have been absorbed into the baseline. 

• The discrepancy report backlog has been reduced 51 percent. 

• The discrepancy report density has been reduced to a record low of 69 per million lines 
of code. 


Metrics have been a powerful force in enabling us to properly plan tasks and allocate resources. 
Our Simulations Applications Load BuUd team planning activities provide an example (figure 
8). The team’s mission is to accept software from over 100 different sources and bm t e 
software into an executable "load". In 1988 the team was producing 880 loads in a year with 
a projected increase in workload as depicted in the lower dotted line in figure 8. As process 
improvement investments began to mature, we were able to absorb an increasing work load and 
in each succeeding year we have reestablished our goals. We now project that the load budd 
group will be able to produce 4891 loads in 1993 - with the same staffing level as in 1988 ! The 
load build group has been able to achieve these productivity levels by significantly improving 
the quality of their processes (figure 9). Figure 9 illustrates the effect of measurmg, monitoring 
and goal setting. The team experienced a 16% error rate in 1989, and established a goal to 
reduced the rate by 25 % in 1990. The team exceeded the goal and developed new goals m each 
subsequent year. The error rate for a very complex process is now less than five percent. Each 
department in Paramax has had similar success. These accomplishments have enabled us to meet 
the constantly increasing volume of Space Shuttle software work while operating within u get 
and meeting more than 98 percent of our schedule commitments. 


SEl^92-004 page 143 



SEL-92-004 page 144 


ACTUAL & PROJECTED RATE OF LOAD BUILD INCREASE 

Goal i Increase Productivity Without an Increase in Personnel (Automate Procedures, 
Streamline Processes, Training, Front End Diagnonstics ) . 

Thousands 



Actual Builds 
Projected (June 1990) 
Projected (Jan. 1991) 
Projected (June 1992) 


0.88 

0.88 

0.88 

0.88 


1.72 

1.72 

1.72 

1.72 


2.657 

2.542 

2.657 

2.657 


3.882 

3.111 

3.417 

3.882 


3.446 

3.93 

4.445 


3.614 

4.237 

4.891 
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11. Business Results 

Our TQM efforts have resulted in our becoming the software supplier of choice for NASA at 
the Johnson Space Center. We have steadily increased our share of the JSC software 
engineering market by enhancing our current contracts with momentum business, and have won 
six of the last seven contract competitions in which we were a bidder. We believe there is a 
direct correlation between our quality progress and our business success. 

12. Quality Results 

Paramax has received recognition from our customers, NASA and external organizations for our 
TQM approach, deployment and organizational results. Some of the major awards and honors 
we have received are the following. 

• Organizational Excellence Award, the top national honor of the Association for Quality 
and Participation (1990) 

• Johnson Space Center Team Excellence Award (1992, 1991, 1990) 

• NASA Excellence Award finalist (1990, 1991, 1992) 

13. Summary 

Paramax performs one of the most complex technical tasks in our industry. Our success in this 
regard is directly attributable to our effective implementation of a comprehensive Total Quality 
Management program that places the highest priority on preventing errors. Our extensive 
metrics and measurements process has been the cornerstone of our quality process and has 
allowed us to set clearly defined, quantifiable goals; monitor our progress toward achievement; 
and make data-driven decisions. 
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The Role of Metrics 
in a 

Software Intensive 
TQM Environment 


Charles B. Daniels 
Para max Space Systems 
December 2 . 1992 


• Background 

• The Measurement Process 

• Team Excellence 

• The Objectives Matrix 

• Operational Results 

• Future Directions 
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Paramax Space Systems 
Software Engineering Role 


• Software products, services and support 

- Space Shuttle/Space Station Operations 

- Mission Control Center Upgrades 

- Space Station Control Center Development 

- Software product & quality assurance at JSC 

- Space Shuttle preflight evaluation 


Initial Environment 


STSOC Facilities 
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Software Complexity 
Constraints (ScMuk Pari 
Variety of Environments 
Testing MgorfDIfficufty 
Software Maturity 
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Multi-process Test Bed Environment 


• Mature software evolution 

- Mission Control Center 

- Shuttle Mission Simulator 

• New Capability Development 

- FADS 

• New Methodology Development 

- Information Systems 


Software Engineering 
Management Focus 


Achieve increasing levels of: 

• Reliability 

- Productivity 

- Responsiveness 
Accomplished through: 

- Baselining existing processes 
> Developing action plans 

- Measurement and analysis 

- Teamwork 



SEL-92-004 page 149 




In The Field of Software Engineering 


The Role of Metrics 


• Tells us where we are (process control) 

• Tracks progress period-to-period 

• Allows management to make data-driven 
decisions 

• Allows meaningful goals to be set 


Metrics are decision indicators. 



SEL-92-004 page 150 







SEL-92-004 page 151 










- Team-building 


mi 

*gr 

«ir 

ill! 
IS | 

mi 

HU 


! 1«3 

sen 


889 ^smiSIIIISS 

M} ••••Kg» 8 <K«Kfl| 

w? 33333333333333 

«*; 33333383333338 

It3 -«*g*J***sg-'*5 

BJ3 -•S|’6 8 S !!B ! BR I 
889 •’Mmbmmbmj 
««a **i8fiiii|t!Ki| 

t!) sC# 8 B S55088 BB l 
M3 ••S|i8»l«r h * 
585 ••«im»i|«| 


88585RMRM8 5 R 8 1*3 

It 5*1 

111381111118 3 •# 
133333383813 a** 

ft 3"l 

ssm a"| 

i«Mi M nz*» 5"| 

Er « * a*? 

3"J 

••8R'****"' , *ft 3"* 



SEL-92-004 page 152 








Paramax Standard Metrics | 

Workflow Performance 

• SR (Change traffic) Closure Index 

• DR (Discrepancy traffic) Closure Index 

• First and Second Level Milestone 

• SR & DR Backlog Management 
Productivity Performance 

• Effort Expended for SRs, DRs, and Testing 

• Resource Allocation 

• S/W Engineering Productivity (KLOC/Engineer) 


Paramax Standard Metrics 

Quality Performance 

• Process Failures 

- Engineering Procedures 

- Requirements Management 
• Supplier Inputs 

- Post Release Production 

• Mean Time to Repair 

- Critical Problems 

- All Problems 

• Backlog Aging - SR & DR 
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Paramax Standard Metrics^ 

Team Building 

• Training 

• Process Improvement Initiatives 

• Staffing Efficiency and Effectivity 

• Critical Skills Backup 

• Information Flow 

• Skill Mix 

• Procedure Currency 



Figim t - Iton Bacrikace Primary Metric* BoanJ 


SEL-92-004 page 154 




































95 1 38*d KX)-Z6^I3S 








Our Challenge] 

• Develop a better understanding of processes 
associated with the "product" aspect of 
software engineering. 

• Develop metrics to measure the performance 
of "product" processes. 

• An ultimate goal of improving the entire 
software engineering lifecycle. 
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